Segmental optical phonetics for human and machine speech processing

That talkers produce optical as well as acoustic speech signals, and that perceivers process both types of signals has become well known. Although perceptual effects due to audiovisual speech integration have been a focus of research involving the visual speech stimulus, relatively little is known about visual-only speech perception and optical phonetic signals. This knowledge is needed to exploit optical signals for applications such as synthetic artificial talking heads and audiovisual ASR. One important practical concern is the wide variation in performance among individual visual perceivers and talkers. This paper focuses on variation in visual phonetic perception, phoneme distinctiveness and word recognition. The paper also introduces a project linking optical phonetics, speech kinematics, and perception.